On Minimum Violations Ranking in Paired Comparisons 
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Ranking a set of objects from the most dominant one to the least, based on the resuhs of paired 
comparisons, proves to be useful in many contexts. Using the rankings of teams or individuals 
players in sports to seed tournaments is an example. The quality of a ranking is often evaluated 
by the number of violations, cases in which an object is ranked lower than another that it has 
dominated in a comparison, that it contains. A minimum violations ranking (MVR) method, as 
its name suggests, searches specifically for rankings that have the minimum possible number of 
violations which may or may not be zero. In this paper, we present a method based on statistical 
physics that overcomes conceptual and practical difficulties faced by earlier studies of the problem. 



I. INTRODUCTION 

A paired comparison of a set of n objects 
{Oi, O2, ■ . ■ , On} is conducted by forming a pair of ob- 
jects from the set and then deciding the better (stronger, 
preferred, more dominant, etc., depending on the con- 
text) one between the pair. If aU (2) possible pairs are 
compared, the comparison is called complete, and incom- 
plete otherwise. In either case, it is commonplace to at- 
tempt to rank the objects from the "best" (ranked high- 
est, at first place) to the "worst" (ranked lowest, at n-th 
place). An example is competitive sports. In a typical 
sports match, two teams or individuals play against each 
other, and a "winner" and a "loser" are pronounced at 
its conclusion. It is well known that many sports, in- 
cluding soccer, tennis, and American collegiate football, 
maintain an official ranking of teams or players based on 
their performances in those matches. These rankings are 
then often used to seed tournaments Q. 

We can find examples in other fields as well. In biol- 
ogy, acts of aggression or submission of an animal towards 
another motivates the study of "dominance hierarchies" 
within the herds or communities of the animals An- 
other instructive example is a certain form of a decision- 
making process. Imagine a city council that, due to a 
limited budget, can only approve a handful among all the 
development projects that its constituents are demand- 
ing (improving schools, building new hospitals, repairing 
highways, and so forth). Assume that the council has 
conducted paired comparisons of the projects so that it is 
known which the constituents prefer, given two projects 
at a time 25]. What would be the optimal selection of 
projects to pursue? If the council is able to rank the 
projects in such a way that no project is ranked lower 
than another project to which it was actually preferred 
to, it may just approve the projects starting from the top, 
possibly escaping the blame for "messing up" priorities. 
However, this may not always be possible because, in 
paired comparisons, preferences are not necessarily tran- 
sitive. Letting Pi > P2 mean that "Project Pi is pre- 
ferred to P2" 12^, we could certainly imagine a cycle of 
preferences, e.g. school > hospital, hospital > bridge, 
and bridge > school. (The reason for this is that the cri- 



FIG. 1: An illustration of the multiplicity of minimum viola- 
tions rankings. Three objects (1, 2, 3) are depicted as labeled 
nodes, and the rest of the system as a gray box. An arrow 
points from the winner to the loser in a paired comparison. 
In (a) and (b), we see that two unbeaten objects may change 
places freely at the top, without affecting the number of vi- 
olations. However, being unbeaten does not guarantee being 
at the top, as we see in (c) which has the same number of 
violations as (a) and (b) have. 



teria of comparison may depend on the specific objects 
being compared 1^.) If there is such a cycle, any ranking 
will inevitably show at least one instance of a project be- 
ing ranked lower than another that it has been preferred 
to. This is called a violation or an inconsistency of 
a ranking (5|. The next wisest strategy, then, would be 
to come up with a ranking that has the minimum num- 
ber of violations, which is called a minimum violations 
ranking (MVR) of the paired comparison. Since the 
number of violations cannot be negative, a zero- violation 
ranking is a special case of MVR. 

Therefore, the advantage of having an MVR is clear 
(i.e., it is the best that one can do to minimize messed- 
up priorities), with its meaning very well-defined. Prac- 
tically, however, the application of this method to real 
situations has been limited for a couple of reasons. First, 
the exact algorithm that finds an MVR of a paired com- 
parison is slow (0(n^)) This is reflected in the fact 
that, despite a century-long obsession of dedicated fans 



with figuring out the "best" football school, only recently 
has an MVR been found by Coleman for the American 
college football schedule that is composed of more than a 
hundred teams '3| . Some of the previous studies on MVR 
have tried to introduce some heuristic methods to speed 
up the search, but have had to make sacrifices in the final 
results 0, IE S 111 ■ Second, the issue of the multiplicity of 
MVRs has been ignored in these studies. For example, if 
there are two "unbeaten" objects (i.e. those not have lost 
in any comparison), then it cannot matter to us which 
one is placed first and which second, since both cases 
will give the same number of violations. This is shown 
schematically in Fig. ^ (a) and (b). In the figure, three 
objects (1, 2, 3) are drawn as labeled nodes that sit at the 
top three spots, and an arrow is drawn from the winner 
to the loser of a paired comparison. To make matters 
more complex, being unbeaten may even not guarantee 
being at the top among other unbeaten ones, as we see 
in Fig.HKc). 

This raises the question of fairness: going back to our 
example of a city council, there can now arise a situa- 
tion where Pi should get approved over P2 according to 
one MVR, but the opposite is true according to another 
MVR. The city council is back in a difficult position of 
having to decide on one, despite the fact that both MVRs 
are equally qualified. Is there a way to break out of this 
situation, without adding another ad hoc criterion? Here 
we propose a natural solution, which is to treat all MVRs 
as equally probable, and take the mean of each object's 
ranks. We then arrange the objects in the increasing or- 
der of these mean ranks to obtain a final standing. For 
example, assume that the three rankings shown in Fig. 
are the only possible MVRs of a some paired compari- 
son. Then the mean ranks of the three objects (1,2,3) 
are {1 -I- 2 1, 2 l-f 3, 3 3 2}/3 = {1.33, 2, 2.67}, 
from which we obtain the final standings {1,2,3}. Al- 
though this looks identical to Fig. QJa) , they must not 
be confused — Fig. QJa) merely represents one specific 
MVR (among many) of the paired comparison. 

While being straightforward and fair to all MVRs as 
requested, this solution seems to ask us to find all pos- 
sible MVRs. Assume that m pairs have been compared. 
Since there are n! different rankings of n objects, there 
will be on average n\/m rankings that show a particu- 
lar number of violations. This can be a huge number, 
even for a common data set: in 2004, the American col- 
lege football schedule consisted of 117 schools playing 622 
games, which gives 117!/622 ~ 10^^°! 

An exhaustive search through such a large state-space 
is therefore clearly impossible, and so we are in need of 
some sampling technique. As a matter of fact, we find 
ourselves in a very familiar territory: we are trying to 
average certain observables (ranks of objects) over a set 
of a large number of microstates (rankings), given a con- 
straint (number of violations). These are the features of 
a typical combinatoric problem that can be formulated 
as a statistical physics model 0, 0, , and that is 
precisely what we will do in the following section. 
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FIG. 2: There exists a natural mapping between (a) the 
n-space and (b) the A'^-space. Each ranking in n-space is 
mapped to (^) rankings with identical relative standings in 
the A''-space. The ranks in the n-space are linearly scaled into 
(c) the mean ranks in the A'^-space according to Eq. Q 



II. DEFINITION OF THE MODEL 

We represent the results of a paired comparison of the 
set of n objects as an rt x n matrix A, called the ad- 
jacency matrix. Its element Aij is equal to 1 if j has 
beaten i in a comparison and otherwise. In an incom- 
plete comparison, there exist pairs of objects for which 
Aij = Aji — 0. However, we require that the set of ob- 
jects cannot be divided into two sets Si and S2 such that 
Ay = and A.ji for all Oi G Si and Oj £ S2. This 
is necessary because if such a division is possible, objects 
in 5*1 and S2 are completely independent, and therefore 
it would be nonsensical to try to rank them on a single 
linear scale. 



A. Expansion of the State Space 

The definition of a ranking of n objects we have used so 
far is an ordering of the n objects from the first place to 
the last (n-th). More precisely, we assign to each object 
an integer rank (1 < < n), such that the ranking 
r = {ri,r2, ■ ■ ■ ,r„} is a permutation of {1, . . . ,n}. (Al- 
though we do not allow ties in a ranking, they can be 
produced in the final standings if the mean ranks of mul- 
tiple objects are equal.) We can also visualize this as n 
objects occupying n ordered slots, which we will call the 
n-space. This is illustrated in Fig. Ol^a) for n = 3. It 
will be shown later that it is more convenient to work in 
a larger TV-space, where the same n objects occupy n 
slots among N {N 3> n) ordered slots. The ranking in 
the A'^-space is similarly defined as s = {si, S2, ■ ■ ■ , s„}, 
where Si is the label of the slot occupied by object i. This 
is illustrated in Fig.[21[b). 

We can define a natural mapping between a ranking in 
the n-space and rankings in the #-space that have the 
same relative standings of the objects (i.e., in terms of 
being higher or lower than others), hence the same num- 
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ber of violations. Under this mapping, each ranking in 
the n-space corresponds to exactly = N"'/nl rank- 
ings in the A''-space. Therefore, Fig. (zfa) gets mapped 
onto Fig.llJb). 

Now, consider an object ranked r in some ranking in 
the 71-space. Its mean rank s{r), averaged over the map- 
ping of the ranking in the A^-space, is given by 



N 



(N- 



N 



1 



(r-1)! 
C{N,n,r). 



TV" 



(1) 



Numerical studies strongly suggest that C{N,n,r) = 
l + 0{n/N) — > 1 for > n. If this is true (and for all the 
examples that we will be presenting in the final part of 
this paper, it is), Eq. Q tells us that s(r) is identical to r 
up to a constant prefactor, i.e. s(r) = r-{N/n+l) = r-C. 
Assume that we have a set i? = {r", r '',...} of rankings 
in the n-space (e.g., of all MVRs), and that we are trying 
to calculate the mean rank rl of object i over i?, which 
is given by 



+ ■■■) 



\R\C 



(2) 



Thus we can conclude that calculating rl over R is 
equivalent to calculating si over the corresponding rank- 
ings in the A'^-space of R under the mapping. See 
Fig. I2tc). We will be working exclusively in the A^- 
space from now on. 



B. Statistical Mechanical Formulation 

It is well known that the canonical ensemble formal- 
ism greatly facilitates the calculation of physical quan- 
tities in a statistical mechanical system such as that of 
gaseous particles, while giving identical predictions as the 
microcanonical ensemble formalism. The essence of the 
canonical ensemble formalism lies in one being flexible on 
which microstates to include in evaluating the partition 
function; typically, all states of different energies E are 
allowed, whereas in the microcanonical formalism only 
states with a specific energy are allowed p^] . 

We will formulate our problem in an entirely analo- 
gous fashion: we include all microstates (rankings) of 
any number of violations (i.e., not just the MVRs) to 
form an ensemble. As usual, the first step is to assign 
the probability P(s) to each ranking s. The fairest, most 
unbiased way to do this is to maximize the entropy 



objects' ranks {si, S2, ■ • ■} and the number of violations 
y(s), which can be written as 



y(s) = - *^)^''^- ^ E A 



(4) 



where 6(sj — Si) = &ij is the Heaviside stepfunction. 
Therefore, the constraints are given as 



s" = X s,F(s) and y = X V{s)P{s). 



(5) 



Using the Lagrange multiplier method on S, we get the 
following solution for P{s): 



Pis) 



3+/3V(s) 



,-(ffo(s)+Hi(s)) 

z ' 



(6) 



where a — {ai, 
pliers, and Hq i 
are called Hamiltonians. The normalizing factor Z is the 
usual partition function, given by 



an} and /3 are the Lagrange multi- 
d ■ s — — J2i c^iSi and H2 = —jSV 



(ffo(s)+ffi(s)) 



(7) 



Since we stipulated that all rankings with the same 
value of V be given equal probability, P{s) of Eq. © 
can be a function of V only. Therefore, all values of 
a are actually equal to 0. They are formally retained 
in Eq. © because they will serve as the differentiation 
variables when calculating from Eq. @ : 



Si = X SiP{s) 



1 dZ 

Z dai 



5=0 



dF 

dai 



(8) 



5=0 



where F ^ — In Z is the free energy. On the other hand, 
(3 will be a negative real number, as we are trying to 
minimize (i.e. discourage the formation of) violations. 
Furthermore, the MVR limit is given by (3 = —00. 

The model is now fully defined. Before we proceed, let 
us briefly discuss the advantages of having this theoret- 
ical model and solutions to it. The first advantage, as 
mentioned earlier, is that our formulation guarantees a 
fair sampling of the phase space by definition, whereas 
exact or heuristic methods merely attempt to find one 
specific MVR of a paired comparison. The second ad- 
vantage is that, although Eq. © lends itself to Monte 
Carlo simulations in a straightforward manner (if it can 
easily be done for (3 — —00 at all) for numerical results, 
having an analytical solution by which we can calculate 
s for any paired comparison would be more convenient, 
possibly requiring no simulation effort at all. 



5 -XP(s)lnP(s), (3) III. PERTURBATION EXPANSION 

s 

subject to certain constraints given by the mean val- We are not currently aware of a general, closed-form 

ues of the observables [TEI IT^ . Our observables are the solution for Eq. {Tj), and therefore for Eq. ©. However, 
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in statistical physics there are a number of sophisticated 
approximation methods available, such as the perturba- 
tion expansion technique which forms the basis of the 
solution developed here. 

First, we re-express the partition function Z: 



-Hi 



Zn 



= Zo^Po(s)e-^i =Zo(e-^i), 

S 

(9) 

where Zq — X^s'^^^"' ^^"^ (' ' ' ) denotes the average of 
a quantity in the ensemble of Hq, in which ranking s is 



given a probability Pq (s) 



"/Zo = e" VZo. Thus we 



are treating Hi as the perturbation to Ho. 

Hq has a very simple meaning: object i is ranked at 
Si with probability Pi{si) oc e"'**', independent of any 
other object. This is also reflected in the fact that Zq is 
a product of independent functions of ai : 

n N 

Zo = ^e-^«(^) = ^eS."-^' = n E e""'' 



i=l Si=l 



n 



e"'(l 



n^(«^)(io) 



i=l 



Therefore, we have 



P^{si) 



z{ai) 1 - e^ 

Now, with Hi — —(3V, Z and F become 



Z^Zo{en = Zo(l + f:^{V')), 



1=1 



(11) 



(12a) 



F 



\nZo - ln( 1 + 5] ^{V) )=Fo + Fi 



(12b) 



Therefore, Eq. (|SJ| can be written as 



dFo _^ dFi 
1 dz{ai) 



z{ai) dui 



dFi 
da. 



N + 1 



+ A, 



5=0 



(13) 



We can understand this intuitively. The first term (N + 
l)/2 comes from Hq, in which an object can assume any 
rank among 1, . . . , iV with uniform probability 1/A'^ (with 
a = 0). {N + l)/2 is simply its mean, the midpoint. The 
second term is, then, the "shift" from this midpoint, 
caused by correlations with other objects via V. 



A. Evaluation of {V') 

Eq. (|12|l tells us that in order to evaluate Z or F, 
we need to calculate (V^) = ((X^ij ^ijf) (with a tacit 
understanding that the summation is done over (i, j) for 
which Aij = 1). 

Let us look at some examples. First, consider ly): 



(14) 



Since 0^ is equal to 1 when Si < Sj and otherwise, 
{Qij ) is simply equal to the probability that Si < Sj . We 
can easily calculate it using Eq. (|ll|l : 



Si<S. 



P,{Si)Pj{Sj) 



N N 

E 

Si— 1 s 



E PM)Pj(' 



l-e"' -e^"^ (l -e"'(e"^ -h e^"- (1 - e"0)) 



(1 - e^«')(l - e^"^ )(l - e«'+"j 



(15) 



The validity of this can be checked quickly by noting its 
limiting value for ai,aj — > 0, which is ^+2^ — ^- This is 
as expected, since if we randomly put two teams at any 
position with uniform probability (this is what a — > 
means), there will be ^ chance that one will be ranked 
higher than the other. 

Next, consider (V^) = J2ijJ2km{^v^krn)- We have 
three types of products QijQkm, depending on the rela- 
tion between and {k,m): 



1. = {k,m) so that 6^9^™ 

8 is cither 1 or 0. 



= 9fj- = Qij, since 



2. and {k,m) share no index, so that i ^ k,m 
and j ^ k, m. We shall call this a disconnected 
product. 

3. and (fc, m) share one index, for example j = k 
and i ^ m. We shall call this a connected product. 

Terms of the first kind will sum to (V). In the second 
case, since the positions of i and j are independent of 
those of k and m, we can easily see that {QijQkm) ~ 
{Qij) (0km)- The third case is more interesting, because 
{QijQjm) does not factor like the previous case since 9^^ 
and Qjm are "connected" via j. We can see this explicitly 
by calculating its average in a manner similar to Eq. H15|l : 

(ey-ej™) = P^is^)PJ{sJ)Pm{s^) 

Si<Sj<Sm 

Sj N N 
= E E E P^i^^)PJisJ)Prn{s„^). 



Si — 1 St— 1 Sm—Si 



(16) 



Its limiting value for ai,aj,am ^ is i not j, as it 
would have been if the product factored. 
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Therefore, (1/^) can be written, schematically, as 



disconnected 



connected 



(17) 



The reason for stressing the distinction between discon- 
nected and connected products will become clear in later 
sections. 

Here, let us study the general implication of 8^ on 
. Assume that we have applied the rule exhaustively 
to every term of . Then will have become a linear 
combination of all possible products of 1, 2, . . . , ^ distinct 
0s, namely, 



fe=i 



aik(t)k 



(18) 



where <f>k is the sum of all possible products of k distinct 
0's, and {a/fc} are appropriate integer coefficients. For 
example, can be written as 



y3 ^ (9^2 + 034 + 656 + •■•)' 

= 6(612934656 + •••) 

+6(612634 + 612656 + 634656 
+ (6i2 + 634 + 656 + ---) 



+ •••) 



03,303 + a^a't'^ + 03,1(^1. 



(19) 



The meaning of aik is clear: it is the number of ways 
of selecting one 6 from each V such that the selection 
of size I includes k distinct types of 6s. Now, consider 
a particular set of k distinct 6s. The number of ways of 
choosing I elements from the set, without the constraint 
that each of k types must be present at least once, is 
simply k X k X ■ ■ ■ ^ kK This is also the sum of the 
number of ways that the selection contains 1, 2, . . . , fc dis- 
tinct types of 6 from the set. This leads to the relation 
J2m=i (m)"'™ = ^' O'^' equivalently. 



aik 



k-l 

E 

m— 1 



(20) 



Multiplying each side by z''/l\ and summing them over 
I = 1, 2, . . . , 00 gives the recursion relation for its gener- 
ating function gk{x) = X^i^i 



k~l 



fc(z) = (e'=^-l)-^ 



9m{z). 



(21) 



This implies gk{z) — (e^ — l)'^ by mathematical induction, 
with the initial condition gi{z) = X^i^i z''ai^i/k\ = (e^ — 
1) and an = 1. This is the solution to the same result 
previously reported by us in p^ . 



Since = V, Z of Eq. (|12a|l now becomes 



00 ^/ r 00 



7 Rl 



1=1 



1=1 



k=l 



1 + EE 



/3' 



-aik 



fc=i '-/=i 

00 



i+E^'<<^')- 



.) = l + ^(e'^-l)'=(</)fe) 
(22) 



k=l 



1=1 



This looks as if we have replaced {V'') and /3 in Eq. 1)12(1 
with {(f>i) and 77 = (e'^ — 1). But what we have here is more 
than a facelift: thanks to 77, the evaluation of ((f>i) up to 
any finite / gives us an infinite-oidev expansion in terms 
of (3. This is analogous to the "partial resummation" 
in field theory. The MVR limit under this change of 
variables is rj — —1. 



B. Expansion of the Free Energy 

Using Eq. , Fi of Eq. 112bl) can also be written as 
a power series of rj: 

/ 00 \ 00 

i^i = -ln l + ^r;'(0,)) =-E^'/'' (23) 
^ 1=1 ' 1=1 

where its coefficients // can be determined by the Tay- 
lor series 



- ln(l + 2;) = -X -f y - y 



For example, the first three /'s are 



(24) 



/i = 
h = 



2)- 



h = (03)-(02)(0l) + 



^1? 



(25) 



Since 0/ is a sum of products of 6s, this tell us that Fi 
is a linear combination of products of their unperturbed 
averages. For instance, it has terms made up of powers 
of (6y): i.e., (6ij) from ((/)i), (6^)^ from ((/)i)^, and so 
forth, which all add up to a quantity ■!/'[6y]. We will call 
this quantity the contribution of 6^^ to the free energy 
Eq. 123): 



V'[6.,] = -r;(6.,) + ^(6.,)2 - ^(e.,f 



(26) 



Similarly, we can define V'[6ij6fcm] (6ij ^ Qkm), which 
is the sum of the products of averages of products involv- 
ing both Qij and 6^,^ . These are trickier than Eq. H2t)|l : 
we have {QijQkm) from ((^2), (6„)(6fem) from (0i)^/2 




FIG. 3: Diagrams representing O and their products, (a) 
Qij is visualized as an arrow pointing to i from j. Products 
of multiple Os are drawn as juxtaposition of arrows, (b) A 
connected product QijOjm is drawn as a connected diagram, 
(c) A connected diagram representing Qik&kjQikQim- (d) A 
disconnected diagram representing OikQkjOim- 

in /2, {QtjQkm){Qij) from (02) ((/'I ) in /a, and so forth. 
These add up to give 

+ 7?'((eyefo„) - (ey)(efe™))((ey + Okm)) 

+ • ■ • . (27) 

Before we proceed any further, let us introduce the di- 
agrammatic representation of products of Os, shown in 
Fig. |3| Here, a single Qij is drawn as an arrow pointing 
to i from j (Fig. |SJa)). A juxtaposition of two or more 
arrows will be understood as a product of the correspond- 
ing Os, such as Fig.|3Jb) which stands for QijQjm- Note 
that the diagram for the connected product Oy O^m con- 
sists of a single component. In graph theory parlance, 
such a diagram is also called connected. As a matter of 
fact, a connected product implies a connected diagram 
and vice versa in this representation scheme. Examples 
of connected and disconnected diagrams (i.e., products) 
of more than two Os are shown in Fig.|3Jc) and (d). 

Therefore, evaluating the free energy Fi means calcu- 
lating tp of all possible diagrams, and Eqs. (^5)1 and 
are such calculated up to 0{r/^). However, any power 
of (Qij) can only come from Taylor-expanding the log- 
arithm of T]{Qij) contained in the 77(^1) of Eq. H22|l . we 
can easily see that has to be 

V'[0.,]--ln(l + rKe.,)). (28) 

Taylor-expanding this to 0(?7'^) indeed gives Eq. H26() . 

The situation is similar for iplQijQkm]- It could have 
only come from Taylor-expanding — ln(l-|-77(Oij-|-OfcTO)-t- 
rf{QijQkm)) of Eq. 1(23), but this also produces powers 
solely of (Oy) or (Qkm), which we now know are '(/'[©jj] 
and ^[Qkrn]- Therefore, we have 

V'[0y0/cm] = -ln(l + r?(Oy +Ofc„)+772(OyOfc™)) 
-^[O,,] - V[Ofo„] 

^ r 1 + l^{e^j + Qkm) + (Ojj Ofa„) ' 
'1 (i + ?7(ey))(i + ?7(efe™)) 

(29) 
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FIG. 4: Connected diagrams composed of up to three arrows. 



Now, assume that OyOfem is disconnected, i.e. 
(OijOfon) = (Oy)(OA;m). Then ?/'[OyOfc,„] vanishes. 
This means that only connected products of two Os con- 
tribute to the free energy. In fact, this is generally true 
for products of any number of Os: only connected prod- 
ucts (diagrams) make nonzero contributions to the free 
energy. Therefore, Fig. ^c) contributes; Fig. I^J^d) does 
not. 

Now we have an iterative prescription for calculating 
Tp of any diagram: from the negative of the logarithm of 
1 plus the terms that represent itself and its subgraphs 
in Eq. (|22|l . subtract the i/^'s of the subgraphs. As an 
example, let us use this rule to evaluate ^^[QijQkm^iq]- 
The subgraphs ofOy O/cmO;g areOy , Qkm, Qiq, QijQkm, 
QkmQiq and 0/gOy . Therefore, 

[0.yOfc,„0/g] 

- ln(l + r;(0,y + Qkm + Qiq) + ??^0,y Ofo„ 

+ 0fo„0,5 + QlqQij) + Tf{QijQk.nQlq)) 
-V'[©y0fem] - iA^kruQlq] - ^[0/,0,,] 

-V'[0,,]-i^[Ofe™]-V[0,,]. (30) 

Again, this term vanishes unless OijOfcmO;, is connected. 

Our strategy is now clear: identify all connected dia- 
grams (composed of up to some desired number of O), 
add up their contributions ■0, and perform the differenti- 
ation of Eq. to obtain s. 



IV. THIRD-ORDER EXPANSION 

Following the strategy stated in the previous section, 
we perform a third-order evaluation of -Fi. Here, "third 
order" means evaluating ip for all connected products of 
up to three Os. (Remember that F\ itself is of infinite 
order in r\ and /3) . Their diagrams are shown in Fig. 21 
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(a) 



(32) 



® 



TABLE I: A simple binary comparison dataset. 



Since Eq. H13() involves differentiation with respect to a 
single ai, we realize that a practical method to evaluate 
averages of the diagrams in ways similar to Eqs. (|15|l and 
(|16|l is to first set all a's to except for ai. This reduces 
all Pj{sj) to 1/iV except for Pi(si), simplifying the cal- 
culation. Now we get two functions of ai for Fig. ^a) - 
one for i at the top and another for i at the bottom — 
for instance. We then multiply each by the number of 
such diagrams present in the data set (the total "losses" 
and "wins" of the object i) to obtain the total shift of 
the diagram to sj. For example, the shift A'°*'*' due to a 
loss by i is 



(b) 



L loss 



da.i 
1 



91n(l + 77(e,,)) 



a.i.cx.j —0 



dai 



Nrj 



cti.aj —0 
(7V2 _ 1)^ 



6(2 + 7?) 



Ui 

-1 N_ 
12' 



6{2N + {N + 

(31) 



This has a positive value for r] < 0(/3 < 0), which is 
anticipated since we expect a loss to increase one's mean 
rank. In the MVR limit of 77 = — 1(/3 = —00), it reduces 
to N/12. This is rather a neat result: if an object i has 
Li losses, its mean rank is shifted upwards by Li x N/12. 
By symmetry, a win by i would decrease its mean rank 
by N/12. 

Performing similar calculations for higher-order dia- 
grams gives us functions of 77 not unlike that of Eq. (|31|l , 
which we do not show here. We can, however, easily guess 
them in a couple of cases: the first diagram of Fig. Efb) 
cannot shift the mean rank of the node in the middle, 
since it is being "pushed" equally up and down. The 
same goes for all nodes in the final diagram of Fig.^Jc), 
and our calculations indeed give for both cases. 

Next, we will apply the results obtained here to several 
examples of paired comparisons. 
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FIG. 5: (a) The visualization of Tabled] (b) connected sub- 
graphs involving object 1. 
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-0.8 



-0.6 -0.4 



-0.2 



FIG. 6: The normalized rankings of objects depicted in 
Fig.l^a). The perturbation expansion (solid lines) is exact in 
this case, and agrees perfectly with Monte Carlo simulation 
results (shapes). 



shown in Fig. El^b) for object 1. After repeating this 
process for all objects, we obtain the following solutions 
for s: 



Example 1: Simple Tree 

As the first example, take a simple comparison whose 
adjacency matrix is given in Tabled It can be visualized 
as Fig. EJa). We can intuitively see that object 1 must 
be ranked at the top, followed by 2, while 3 and 4 should 
be tied (since they are topologically indistinguishable) 
at last to obtain a minimum number of violations (0). 
In order to use our expansion results, we must count 
the connected subgraphs involving each object, which are 



sT(?7) _ 4r/3 + 25772 + 5O77 + 30 

N ~ 5(77 + 2) (772 +67^ + 6) 

s^{rj) _ 3773 + 2brf + 5O77 + 30 

N ~ 5(77 2) (772 6r? + 6) 

gi(^) ^ irf + 39772 + 80r; + 60 ^ Jl^jj^ 

N 10(r/ -f 2) (772 -h 67? -F 6) N ' ^ ' 

Since we have considered all subgraphs appearing in 
Fig. Ela), these must be exact. An excellent agreement 
with Monte Carlo simulation results for the entire range 





1 

110 (34) 

1110 

11110 



TABLE II: The adjacency matrix for the tournament 5 in [3. 




FIG. 7: The visualization of Table El 

of T] in Fig. 1^1 confirms this. 

A closer inspection of Fig. |H| tells us that the mean 
ranks are {N + l)/2 = N/2 for 77 = 0(/3 = 0) as 
expected, while sT and S2 begin to diverge noticeably 
around 77 = — 0.6(/3 = —0.9). Since objects 1 and 2 are 
indistinguishable on the first order (they both have a 
win- loss differential of 1), this has to be the point where 
second and third-order diagrams begin to contribute sig- 
nificantly. The MVR solutions at 77 = —1 also agree with 
Eq. p. 

Larger data sets will certainly contain fourth- or 
higher-order connected diagrams, and therefore our 
third-order expansion will be an approximation. How- 
ever, it is still interesting to see how valid our third-order 
expansion can be as a true approximation. We study this 
issue with the following examples. 



Example 2: 5-Player Round-Robin Tournament 

A round-robin tournament is a complete paired 
comparison. Ali et al. studied the minimum violations 
ranking problem of the 5-player round-robin tournament 
(with (2) — 10 comparisons) using a couple of heuris- 
tic methods on the nine distinct configurations they had 
identified [^. One of those tournaments (labeled 5 in 
their paper) is given in Table ITll and visualized in Fig. 13 
The correct MVR standings should be {1, 2, 3, 4, 5}. 

Fig. |S1 shows the perturbation and Monte Carlo simu- 
lation results. We see that there exists a wide range of ?/ 
(between —0.8 and 0) in which the perturbation agrees 
with the simulation well within 2% of N and predicts the 
relative standings correctly, whereas closer to 77 = —1 it 
behaves in a haphazard way and gives incorrect stand- 
ings. This latter behavior is certainly due to the missing 
higher-order diagrams. We find almost identical behav- 




I I I I I I I I I I I 

-1 -0.8 -0.6 -0.4 -0.2 

FIG. 8: The third-order perturbation expansion (solid lines) 
and Monte Carlo simulation results (shapes) for the tourna- 
ment in Table |n] There is a wide range of ri for which the 
perturbation gives a correct prediction of standings, followed 
by a region of haphazard behavior near 77 = —1 caused by 
missing higher-order diagrams. 



iors for the eight other tournaments as well: the pertur- 
bation solution is stable and predicts correct standings 
including ties for a wide range of 77, while near ?; = — 1 it 
fails. 

Compare this to the heuristic methods of Ali et at, 
which merely succeeds in finding one specific MVR for 
each the tournament. Furthermore, strangely enough, 
they regard the emergence of "tied ranks" as an undesir- 
able feature of a method in their study. We believe to 
the contrary: it is only fair to give tied ranks to topolog- 
ically indistinguishable players, which our method does 
explicitly. 



Example 3: American Collegiate Football 

As a final example, we consider a large-scale data set 
mentioned earlier, American collegiate football. In the 
year 2004, 117 universities played a total of 622 games in 
the regular season. Ranking of colleges based on the out- 
comes of those games is of particular interest, because 
schools ranked high in the "official" ranking (currently 
determined by a combination of human polls and several 
computer algorithms) get invited to play in prestigious 
post-season bowl games, while the top two schools get to 
play in the national championship game p^ . An invi- 
tation to these games is not only honorable, but brings 
practical benefits to the schools, including much pub- 
licity through the media, financial compensations worth 
millions of US dollars, and an increase in applications 
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il=(sl'-l) 



FIG. 9: The third-order perturbation expansion for 117 
American college football teams in 2004. 

from prospective students jl8j |. 

Due to the small number of games 11) a school 
plays in a season compared to the number of possible op- 
ponents (~ 120), a simple win-loss differential is a hope- 
lessly insufficient measure of a team's strength. There- 
fore, the official computer algorithms as well as the hun- 
dreds of rankings schemes devised by the fans of the sport 
employ various criteria that their inventors or developers 
deem necessary and irnportant in order to assess the col- 
lege teams' strengths 

One often evaluates the quality of an algorithm by 
counting the violations in the final ranking it pro- 
duces [231 • Any MVR will perform the best in this respect 
by definition, but in a recent study of a particular MVR 
that was mentioned earlier was shown to have performed 
as well as or better than other ranking methods in other 
tests as well ^ . Again, this study was done for one par- 
ticular MVR, and so it would be an interesting topic for 
future research to study how s from Eq. (|13|l perform in 
those tests. 

In this paper, though, let us concentrate on our third- 
order perturbation expansion. In Fig. |51 we see that it 
shows a behavior quite similar to the one we have seen in 
the previous example: we have a smooth, well-behaved 
range (though narrower, —0.3 ^ 77 < 0), followed by a 
region of large fluctuations. 

In the smooth range, the number of violations is consis- 
tently 115, short of the 53 found by Coleman. Neverthe- 
less, it is interesting that the violation decreases rapidly 
to about a third of 311 (half the number of games) ex- 
pected at 77 = as soon as 77 is tuned away from it (at 
77 = —0.001), and stays that way over the entire region. 
The relative standings of the teams also change very lit- 
tle, suggesting that they may still reflect the correct ones 




0.1 0.2 0.3 0.4 0.5 
differences in (mean rank)/N 



FIG. 10: The normalized distributions for the differences in 
s/N between object pairs across violations and non-violations, 
for rj — —0.3. This shows that violations occur between object 
pairs that are generally closer in their mean ranks. 



to some degree. If this is the case, then the violations 
must be occurring between teams that are closer in s 
so that adding more diagrams are likely to swap these 
pairs first, causing only limited changes to the standings. 
In Fig. ^1 we show the distributions of the separation 
in s/N between the 115 pairs of teams across violations 
and the 622 — 115 = 507 pairs that are not, for 7; = —0.3. 
We clearly see that l/N are closer across a violation on 
average. Actually, by directly swapping the positions of 
pairs of schools across a violation in the increasing order 
of their separation in s/N, we were able to reduce the 
number of violations by another 20 — 17% of the origi- 
nal 115 — down to 95. (In order to lower it further, we 
need to swap objects in a more complicated order. In this 
sense, a higher-order expansion is precisely what tells us 
how to do that.) 



V. SUMMARY AND DISCUSSION 

In this paper, we have formulated the problem of min- 
imum violations rankings in paired comparisons as a sta- 
tistical physics model, analogous to the canonical ensem- 
ble theory of systems of particles. This formulation en- 
sured that all configurations with the same number of 
violations were assigned equal statistical weights in the 
ensemble, overcoming the conceptual difficulties faced by 
earlier studies of the problem. 

We then developed a diagrammatic perturbation tech- 
nique for evaluating the free energy F, from which we 
could obtain the mean ranks s of each object via differen- 
tiation. Making use of the property Q'^{x — y) = Q{x — y) 
of the Heaviside step function, we were able to find an it- 
erative prescription for evaluating inflnite-order contribu- 



10 



tions of finite diagrams (i.e., partial resummation) . The 
fact that we only needed to consider connected diagrams 
greatly facilitated the expansion, and a third-order per- 
turbation expansion was carried out explicitly. 

We were able to use these results on any type of paired 
comparison data set, having only to count the number of 
connected subgraphs it contained. We studied three ex- 
ample data sets. In the first example, the perturbation 
expansion had to be exact, and we confirmed this by 
comparing it to a Monte Carlo simulation. In the latter 
two examples we were interested in the performance of 
the expansion as a true approximation. We witnessed the 
existence of a region in which the perturbation expansion 
was stable and well-behaved. It was followed by large 
fluctuations near 77 = — 1 due to the missing higher-order 
diagrams. The reliability of this stable, smooth range as 
the predictor of the true mean minimum violations s at 
r] = —1 would depend on how high an order we perform 
the expansion into, but already on third order we were 
able to predict the relative standings perfectly for the 
5-player round-robin tournament, and to some reason- 
able degree for the 117-player American college football 
schedule. 

We propose several directions for future research based 
on this work. First, performing the expansion into higher 
orders is always an obvious possibility. Second, develop- 



ing faster algorithms or programming more efficient codes 
for counting the connected subgraphs is desirable. Third, 
a generalization of the theory to cases where Aij > 1 (i.e., 
allowing multiple comparisons or considering the weight 
of a comparison, such as the score difference in a sports 
match) is allowed would be welcome. We hope that these 
efforts will prove helpful in the practical application of 
this method to other large-scale data sets. 

Lastly, we note that the entire perturbation technique 
developed in this paper is a significant improvement over 
the one introduced earlier in in the context of the 
theory of exponential random graphs. We are hoping to 
see more developments in that area as well. 
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